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We investigate various competing paramagnetic ground states of the Heisenberg antiferromagnet on the two 
dimensional star lattice which exhibits geometric frustration. Using slave particle mean field theory combined 
with a projective symmetry group analysis, we examine a variety of candidate spin liquid states on this lattice, 
including chiral spin liquids, spin liquids with Fermi surfaces of spinous, and nematic spin liquids which break 
lattice rotational symmetry. Motivated by connection to large-N SU(N) theory as well as numerical exact diag- 
onalization studies, we also examine various valence bond solid (VBS) states on this lattice. Based on a study 
of energetics using Gutzwiller projected states, we find that a fully gapped spin liquid state is the lowest energy 
spin liquid candidate for this model. We also find, from a study of energetics using Gutzwiller projected wave 
functions and bond operator approaches, that this spin liquid is unstable towards two different VBS states — 
a VBS state which respects all the Hamitonian symmetries and a VBS state which exhibits x \/3 order 
— depending on the ratio of the Heisenberg exchange couplings on the two inequivalent bonds of the lattice. 
We compute the triplon dispersion in both VBS states within the bond operator approach and discuss possible 
implications of our work for future experiments on candidate materials. 

PACS numbers: 74.20.Mn, 74.25.Dw 



I. INTRODUCTION 

In recent years, several geometrically frustrated spin- 1/2 
magnets have been synthesized which appear to not order 
magnetically even at temperatures well below the character- 
istic exchange couplings. Among these are the quasi-two- 
dimensional (2D) triangular organic material k-BEDT(CN)3,' 
the kagome lattice herbertsmithite,^ and distorted kagome lat- 
tice volborthite,^ and the three-dimensional hyperkagome lat- 
tice magnet"* Na4lr308. A large class of these magnets ap- 
pear to exhibit gapless spin liquid behavior down to very 
low temperatures, leading to the exciting possibility that 
they may possess exotic ground states with fractionalized 
excitations lOi 1 1 , 12, 1 3 Qtjjgj-g among these have been pro- 
posed to weakly order into singlet valence bond solid (VBS) 
states which break lattice symmetries ! As yet, there is 
no clear picture of what combination of geometric effects and 
spin interactions will lead to spin liquid ground states or VBS 
ground states; this necessitates a theoretical and experimental 
exploration of various new lattice geometries as well as pos- 
sible ring-exchange interactions beyond the simplest Heisen- 
berg spin exchange interaction. 

In this paper, we focus on understanding several compet- 
ing ground states of the nearest neighbor S~l/2 Heisenberg 
model 

H = Y,JuS^-Sj. (1) 

on the 2D star lattice, shown in Fig. 1, as a function of Jt/Je 
where Jt and Je are the exchange couplings on the 'triangle 
bonds' and 'expanded bonds'. Our motivation for this study 
is two-fold. First, the recent synthesis of a 'star lattice' or- 
ganic Iron Acetate quantum magnet'^ raises the possibility 
that a S* = 1/2 variant may possibly be synthesized in the 



near future and our results should be applicable to such sys- 
tems. Second, this lattice has a sufficiently different geome- 
try from more commonly studied quantum magnets — it may 
be viewed either as a variant of the kagome lattice or as a 
decorated honeycomb lattice — which allows us to explore 
the effect of this new lattice geometry on possible spin liquid 
physics and valence bond solid phases in quasi-2D systems. 

Motivated by interpolating between this spin- 1/2 SU(2) 
model and an SU(N) generalization at large N which permits 
a mean field solution with fermionic spinon excitations,'^-^" — 
we examine a large number of interesting U(l) spin liquids 
as candidate ground states of the nearest neighbor S — 1/2 
Heisenberg model on the 2D star lattice. Guided by ear- 
lier work on the kagome lattice,— we focus on spin liquid 
states denoted by SL a, ^v, ^dodecagon] where $a,'J'v, 
and ^dodecagon denote, respectively, the "fictitious" fluxes 
seen by the fermionic spinons as they move around an ele- 
mentary plaquette of the lattice: an up triangular plaquette A, 
a down triangular plaquette V, or the 12-site dodecagon pla- 
quette. In terms of the original spin variable, the fluxes on 
the triangular plaquettes correspond to scalar spin chiralities 
of the form Si • S2 x S3, while ^dodecagon is related to an 
operator defined by the twelve spins around the dodecagon 
loop. Depending on the flux values, these spin liquids rep- 
resent gapped chiral spin liquids which break time-reversal 
symmetry,^^ or states with gapless Fermi surfaces of spinons, 
or gapped spin liquids with no broken symmetries. 

From a study of energetics of various flux values using 
Gutzwiller projected wave function numerics for the physical 
case of = 2, we show that a particular gapped spin liquid, 
which we denote as SL[0, 0, tt], which does not break lattice 
or time-reversal symmetries emerges as a favorable candidate 
over a wide range of Jt/Je- This is in striking contrast to ear- 
lier work on the kagome lattice from two perspectives. First, 
as we show, the effect of projection is far more dramatic on 
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FIG. 1 : (Color online) Top: Structure of the star lattice depicting the 
six-site unit cell, the chosen lattice basis vectors ai=2x and 32 = x 
+ \/3y, and the bonds with Heisenberg exchange couplings Jc ('ex- 
panded bonds') and Jt ('triangle bonds'). Bottom: Phase diagram 
of the antiferromagnetic Heisenberg model on the star lattice. For 
Jt/Je <^ 1, the ground state is a valence bond solid (VBS) phase 
(Je-dimer VBS) in which every dimer sits on the expanded links 
connecting neighboring triangles. For Jt/Je ^ 1, the ground state 
is a VBS with an 18-site unit cell (the columnar 18-site VBS). 



the star lattice when compared with the kagome lattice; a nu- 
merical Gutzwiller projection of the mean field states leads to 
a complete reordering of the energies of the candidate spin 
liquids. Second, unlike the kagome lattice case where the 
lowest energy variational state of this form is a spin liquid 
with massless Dirac fermion excitations,^ the SL[0, 0, n] is a 
gapped U(l) spin liquid — we therefore know that it is ulti- 
mately unstable towards spinon confinement at low energies^^ 
unlike the Dirac fermion states whose stability depends on the 
number of fermion flavors Ni^ 

We find that the SL[0,0, tt] state naturally forms strong 
dimers on the 'expanded bonds' for small values of Jt / Je thus 
leading to a confined state, a Jg-dimer VBS, which respects 
all symmetries of the Hamiltonian.^^ For large Jt/Je, numer- 
ical exact diagonalization (ED) studies of this model carried 
out in a restricted nearest-neighbor dimer basis showed signa- 
tures of V3 X \/3 ordering.^'' We argue that another motivation 
to study possible dimer orders that break lattice symmetries is 
that such ordering often appear quite naturally in the large- 
N fermionic SU(N) theory as recognized in the early work 
of Affleck and Marsto n^'*'^" and shown in various other mod- 
els studied recently. ^2*21 Inspired by these results, we consider 



various candidate VBS phases from different perspectives — 
a large-N route, a bond operator formalism, and Gutzwiller 
projected wave function numerics. All of these point to a 
transition to a \/3 x V3 ordered VBS phase for large enough 
Jt/Je ^ 2 — 2.5, leading us to the phase diagram shown in 
Fig. 1. Our result is in broad agreement with the ED study al- 
though the transition point estimated from our work is some- 
what larger than the exact diagonalization (ED) result which 
yields (Jt/Je) ait ~ 1-3; the ED result may, however, may 
suffer from significant finite size effects. We then discuss 
possible routes by which the SL[0,0,7r] state might be un- 
stable towards such VS x a/3 VBS order instead of the Je- 
dimer VBS. We present results for the triplon dispersion in 
both VBS states which can be tested in inelastic neutron scat- 
tering experiments on candidate materials. 

Finally, although the various other interesting spin liquids 
we study do not appear to be energetically viable ground states 
for the nearest neighbor Heisenberg model on the star lattice, 
they have energies which are close to the ground state. They 
might thus be stabilized as ground states by small changes in 
the Hamiltonian, such as further neighbor exchange or spin- 
phonon coupling, or they might be relevant to understanding 
the intermediate energy scale properties or finite temperature 
physics of materials which realize this model. We therefore 
elucidate some of the properties of these spin liquid states. 

This paper is organized as follows. We begin, in Section Ull 
by formulating the mean field theory of the Heisenberg model 
on this lattice in a slave particle description using fermionic 
spinons which we relate to a large-N SU(N) approach. Based 
on this, we classify and study the physical properties of a num- 
ber of candidate spin liquid ansatzes. We next turn, in Section 
lUD to a study of dimerized states on this lattice, and present 
a group theoretic classification of \/3 x \/3 orders as well as 
a large-N justification of specific candidate VBS phases. Sec- 
tion |IV] contains a discussion of the energetics of various spin 
liquid states using mean field theory as well as a Gutzwiller 
projected wave function study and bond operator approaches 
of candidate VBS phases. Section V discusses the various 
ways in which the SL[0, 0,7r] state, which is the lowest en- 
ergy spin liquid state, might be unstable towards VBS order- 
ing as a result of spinon interactions and from a Gutzwiller 
wave function approach. The phase transition between VBS 
phases is described in Section|Vll Section lVlIl contains a dis- 
cussion of the triplon dispersion in both VBS states which we 
think are realized in this model. We conclude with a discus- 
sion about experimental implications in Section [Villi Details 
of various calculations are contained in Appendices A-C. 



II. SPIN LIQUID PHASES ON THE STAR LATTICE 

A. Formulation of the mean field theory 

We investigate the ground state of the S=l/2 Heisenberg 
antiferromagnet on the star lattice, which can be described by 
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the following Hamiltonian, 



H 



(2) 



where i and j indicate the positions of nearest neighbor spin 
pairs. Natural description of this lattice system requires the 
consideration of two inequivalent links, that is, one link lying 
on a triangle (a triangular link) and the other link connecting 
two neighboring triangles (an expanded link). We assign two 
different exchange couplings Je and Jf on expanded and tri- 
angular links, respectively. We can also label a site i by pairs 
(R, n) where R denotes the location of a unit cell and n labels 
the six sites inside a single unit cell. (See Fig.[T]) 

To construct spin liquid states we introduce the fermionic 
spinon operators, (cr or |) to represent the spin opera- 
tor; 



(7) 



Here describing the phase fluctuation of Xij plays the role 
of the spatial components of the U(l) gauge field. Namely, we 
have reformulated the quantum spin model as the problem of 
the spinons strongly coupled to the U(l) gauge field. 

A systematic way of studying the coupled spinon and gauge 
field system is to consider the large-N reformulation of the 
problem extending the spin SU(2) symmetry to SU(N) (with 
N even).^"'^' We let the flavor index a run from 1 to N and 
modify the single occupancy constraint as. 
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(8) 



•^'^ " 2 ^ /Ll<l,<T2/^'^2> (" = x,y,z). (3) 

Since this representation alone contains unphysical local 
configurations enlarging the Hilbert space, we have to impose 
the following local constraint, /|/| + /J/x = 1, to recover the 
physical Hilbert space. Using the fermioninc spinon represen- 
tation of the spin operator, the Heisenberg spin Hamiltonian 
can be rewritten as follows. 



(4) 



Here we have dropped unimportant constant terms. 

To decouple the four fermion interaction term we define 
spin singlet order parameters, Xij = lJ2a iflafj,'^)- ^^er 
imposing the single occupancy constraint using the Lagrange 
multipliers /ij, the mean field Hamiltonian is given by 



In addition, we scale the interaction strength Jy/2 to be 
Jij/N to make each term of the Hamiltonian to be of order N. 
The resulting large-N Hamiltonian is given by 



H 



N 

E 

a,P=l <ij> 
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(9) 



To treat the quartic interactions we perform a mean field 
decoupling by introducing SU(N) singlet valence bond , Xij 

= Tf'^a ifi afj,a)- Assuming the valence bond amplitude 
is a complex number, we obtain the mean field Hamiltonian 
given by 

Hmf --EE JviflahcX*, + h.c.) + iV ^ J,,|Xy| 



a <i]> 



(10) 



cr <ij> 



(5) 



To describe the phase fluctuation of the mean field ansatz, 
we express Xij as Xij = Xij e"*'^ ' which leads to the following 
Hamiltonian, 



U{1) 



E E MflJj,^.,^""'' + h.- 



<y <ij> 



(6) 



In the above Hamiltonian the local U(l) gauge sym- 

metry of the spin Hamiltonian which comes from the local 
conservation of the fermion number is manifest via the fol- 
lowing gauge transformation.S2i^ 



Since the fluctuations of Xij and the average local density 

'k^a ifiafi-a) scalc as \l\fN, we can safely neglect those 
fluctuations in the large-N limit justifying the mean field ap- 
proximation. 

Here we consider the following mean field ansatz Xij = 
\Xij I e*"^'J where |xy | = Xe on expanded links and \x^j \ = Xt 
on triangular links, respectively. We specify the various flux 
patterns inside the elementary plaquettes, i.e., the triangles 
and the dodecagons. The flux inside a triangle for exam- 
ple, is defined in the following way, e**'^ = e'('^'^+'^^''+'^''') 
where {ijk) indicates the three corners of a triangle traversed 
along the counterclockwise direction. The flux inside a do- 
decagon is also defined in the same manner. Since the flux 
inside a closed loop is a gauge invariant object, different spin 
liquid ansatz can be distinguished based on the flux values 
inside the elementary plaquettes. In particular, we use the 
term SL[(1>a, <i>Vj 'i' dodecagon] to represent the ansatz which 
has the fluxes <I>a inside an up-pointing triangle, (l>v inside a 
down-pointing triangle and ^dodecagon inside a dodecagon. 
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FIG. 2: (Color online) The spinon dispersion of the SL[0,0,0] near 
the Fermi level. Among the six bands inside the first Brillouin zone 
we plot the third and four band which are lying close to the Fermi 
energy along the axis. The flat band is lying at the Fermi level 
which touches with the dispersive band at the F point. 



With a given flux configuration we determine \xij\ and fXi 
self-consistently by solving the foflowing coupled mean field 
equations, 



site 

I a 



where iV„tp is the number of lattice sites. 



N 



(11) 



B. Properties of competing spin liquid phases 

In this section we discuss the characteristic properties of 
various spin liquid phases and their instabilities. In particular, 
we focus on translationally invariant mean field states which 
have nonzero \xij\ on every link of the lattice. Extensive 
discussion on possible dimerized phases is given later in 
Sec. ini] As shown in the previous studies about the spin 
liquid phases on the square^i and kagome^^ lattices, the 
inclusion of additional spin interactions can change the 
relative energetics of different spin liquid phases. Therefore 
it is useful to understand the nature of various competing 
spin liquid states which are the potential ground states of spin 
Hamiltonians beyond the nearest neighbor Heisenberg model. 

(a) SL[0, 0, 0] : Uniform spin liquid state 

To describe this state we introduce two real mean field or- 
der parameters, Xe and xt, which lie on the expanded and 
triangular links, respectively. Since the unit cell contains six 
sites we obtain six different bands inside the first Brillouin 
zone. Among the six bands, the third and fourth bands near the 
Fermi level show an interesting structure displayed in Fig. |2l 
There is a flat band lying on the Fermi energy which is touch- 
ing another dispersive band at the zone center, r(k = 0). 
The flat band comes from the existence of the localized eigen- 
states, which occur due to the destructive interference of hop- 
ping amplitudes between the localized units. The flat band of 
the uniform spin liquid on the kagome lattice emerges owing 
to the same reason. However, in contrast to the kagome lattice 



problem, the flat band is lying exactly at the Fermi level on 
the star lattice. 

The flatness of the band at the Fermi energy is not the 
generic property of the uniform spin liquid. There are per- 
turbations which do not break any lattice symmetry but spoil 
the flatness by generating curvature. The third nearest neigh- 
bor hopping is such an example. However, the quadratic de- 
generacy at the zone center is protected by the point group 
symmetry of the underlying unit cell. 

To understand the stability of the spin liquid we derive the 
low energy effective Hamiltonian, which describes the states 
near the zone center, expanding the Hamiltonian up to the 
quadratic order of the momentum k. The procedure for deriv- 
ing the effective Hamiltonian is outlined in the Appendix IA 31 
The resulting Hamiltonian is given by, 

^eff=— / 7^^nk)/leff(k)V(k), 

meff J (2Try 

in which 

hes{k) ={kl + fc2)To - [kl - kl)T, - 2k^kyT^, 

where the Pauli matrix Tj is acting on the two-component 
space of the continuum field (V')^=(?Ai, V'2) which describes 
the two low energy states near the F point. 

The SL[0,0,0] state respects all the space group symmetry 
of the lattice. In particular, if we choose the gauge in which 
Xij=Xt on every triangular link and Xij=Xe on every expanded 
link with xt and Xe being real constants, the action of the sym- 
metry generators on the spin operator. Si, is the same as that 
on the spinon operator, fi c,. Since we consider the low energy 
excitations near the zone center, we focus on the action of the 
point group symmetry on the continuum fields. The Dq point 
group of the star lattice consists of twelve symmetry genera- 
tors and is generated by the two elements, Cil and Ry. Here 
C| means the -I rotation with respect to the center of a do- 
decagon and Ry indicates the reflection about the x-axis. The 
details on the elements of the point group are discussed in 
the SecHnB] 

Under the and Ry, the continuum field i/j transforms in 
the following way. 
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(12) 



Ry : ip - 

meaning all the fermion bilinears ip^T^ip (a = x, y, z) are for- 
bidden by the point group symmetry. Note that the ip^Tyi/j 
breaks time-reversal symmetry as well since it shows sign 
change under complex conjugation. 

Next we consider the fermion bilinear terms that contain 
spatial derivatives. Because the dynamical critical exponent 
is two, the terms with a single spatial derivative are relevant 
and those with two spatial derivatives are marginal pertur- 
bations. Investigating the transformation rule under the Dq 
point group symmetry, it can be easily checked that ip'^Tyip 
transforms as the one dimensional A2 irreducible represen- 
tation and {tp'^Tx^,ip'^Tz^) forms a basis for the two dimen- 
sional E2 irreducible representation.^'* Similarly, the trans- 
formation properties of derivative terms can be determined. 
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At first, the linear derivative term, (dx,dy) transforms as a 
two dimensional Ei ureducible representation. To have terms 
with linear derivatives in the Hamiltonian, the product of the 
fermion bilinear and the derivative must be invariant under 
the point group symmetry operations. Using the decomposi- 
tions of Ei^A2 = El and Ei^E2 = Bi © B2 © Ei, 
we see that every product of fermion bilinears and the lin- 
ear derivative is not invariant under the point group symme- 
try. Therefore linear derivative terms are not allowed. On the 
other hand, we have a second derivative term (d^ — dy,dxdy) 
making a two dimensional E2 irreducible representation. Us- 
ing E2^A2 - E2 and E2^E2 = Ai^A2®E2, we 
see that there is a term following Ai irreducible representa- 
tion, which is nothing but [{2dxdy)Tx + (9^ — dy)Tz]ip. 
Therefore in addition to the isotropic tp^id^ + dy)tp term, 
ijj^ [{2dxdy)Tx + [dx — dy)Tz]i/j is the only term allowed by 
symmetry. Since these terms are already present in the Hamil- 
tonian, the low energy properties of the SL[0,0,0] are not 
spoiled by these marginal perturbations. However, these per- 
turbations add curvature to the flat band. 

Finally, we discuss the effect of the four fermion interaction 
terms on the stability of the SL[0,0,0] state. Though a simple 
power counting shows that the four fermion interactions are 
marginal, they are actually marginally relevant. Recently, 
the effects of the four fermion interaction on the quadratic 
band crossing are studied using the renormalization group 
approach. ^^'^^ According to Ref. the leading weak 
coupling instability leads to the state with nonzero {i/j^Tyi/j), 
breaking the time reversal symmetry. It means that SL [0,0,0] 
state is unstable toward a chiral spin liquid state supporting 
chiral edge states. 

(b) SL[0,G,7r] state 

The SL[0,0,7r] state supports tt flux piercing the do- 
decagons. Even though this state does not break the transla- 
tional symmetry, the mean field description requires doubling 
of the unit cell. Here we consider the doubling of the unit cell 
along the ai direction. For the lattice vectors 2ai and a2, the 
reciprocal lattice vectors are given by 



(a) 



Gi 



The reduced Brillouin zone corresponding to the above recip- 
rocal lattice vectors is depicted in Fig.[3ja). 

For the mean field description of the SL[0, 0, tt] state, we 
have chosen the flux configuration as described in Fig. Ob). 
Since there are twelve sites inside the unit cell, we have twelve 
bands within the Brillouin zone. The mean field spinon disper- 
sion of the low energy bands near the Fermi level is described 
in Fig. |4] This state does not have a spinon Fermi surface and 
shows a gapped spectrum. The lower flat band (valence band) 
is doubly degenerate and the upper band (conduction band) is 
dispersive. 

According to the projected wave function study that is 
discussed in detail later in Sec. lIVBl the SL[0,0,7r] state has 
the lowest ground state energy among the various spin liquid 
ansatz over a wide parameter range. Unfortunately, however. 
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FIG. 3: (Color online) (a) The reduced Brillouin zone (the solid line) 
corresponding to the doubled unit cell along the ai direction. Gi and 
G2 indicate the reciprocal lattice vector corresponding to the doubled 
unit cell, (b) The flux configuration of the SL[0, 0, n] state. The 
twelve-site unit cell is surrounded by a dotted box. We assign -1 (+1) 
for the hopping amplitude on the thick (thin) bond. 
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FIG. 4: (Color online) The low energy spinon bands of the 
SL[0, 0, tt] state near the Fermi level. The lower band (valence band) 
is flat and doubly degenerate. The upper band (conduction band) has 
small dispersion. The locations of the conduction band minimum 
(maximum) are described by nii (Mi). The dispersion is plotted 
along the ky=\/^kx+-^^ line passing the mi and m2. 



the SL[0,0,7r] state is unstable once gauge fluctuation is 
allowed. Since the spinon spectrum has a finite gap, the 
low energy excitations are described by the compact U(l) 
gauge theory. In 2 H- 1 dimension, the compact U(l) gauge 
theory is confining,^^ which means that free spinons with 
unit gauge charge can only make charge neutral bound 
states. In addition, the interaction between spinons can also 
induce various kinds of broken symmetry states. Extensive 
discussion on the instability of the SL[0,0,7r] state and its 
relation with candidate valence bond solid phases are given 
in Sec.[Vl 



(c) SL[^, ^, tt] : A chiral spin liquid state 
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FIG. 5: (Color online) Flux configuration of the ansatz SL[-|, vr] 
and SL[— -I, -1,0]. Hopping along (against) the direction of an ar- 
row induce a phase 7r/2 {-nil), (a) The flux pattern for SL[-|, -I, vr]. 
Counterclockwise motion along an triangular link results in the flux 
■| (b) The flux pattern for SL[— -I, -1,0]. Counterclockwise motion 
along an up-pointing (down-pointing) triangular link results in the 
flux-f (|). 



Next we consider flux phases which possess finite flux in- 
side triangles. A triangle that supports flux breaks time- 
reversal and parity symmetry but preserves the combination 
of them. A convenient way to investigate the time-reversal 
symmetry breaking in spin systems is to consider the expec- 
tation value of the scalar spin chirality operator— defined as 
follows, 

Qjfe = S, • (S, X Sfe). (13) 

Since Cijk is odd under both the time reversal (T) and parity 
operations, the ground state breaks both symmeties when the 
expectation value of the scalar spin chirality operator {Cijk) 
is nonzero. In other words, the scalar spin chirality plays the 
role of the order parameter measuring time reversal symmetry 
breaking. 

Because the unit cell contains two triangles, we can define 
the following two different scalar spin chirality operators, 

Si-(S2XS3)+S4-(S5XS6), 
Cstaggered = Si • (S2 X S3) — S4 • (S5 X Sg), (14) 

where Coniform and Cstaggered are the uniform and stag- 
gered scalar spin chiralities, respectively. To understand the 
symmetry properties of the Cuniform and Cstaggered we have 
to recognize that there are two different reflection symmetries 
on the star lattice. The reflection (Pi) with respect to the axis 
connecting the center of a dodecagon with the mid-point of 
an expanded link (for example, the a axis in Fig. [T2l a)) inter- 
changes the up-pointing triangles and the down-pointing tri- 
angles. On the other hand, the other reflection (P2) about the 
axis connecting the center of a dodecagon with a vertex of a 
triangle, (for example, the A axis in Fig. [TSl a)). leaves each 
triangle as it is. Under the P2 reflection both Cuniform and 
Cstaggered Change their signs. Here we use the term parity to 
indicate the Pi reflection symmetry. Note that the parity oper- 
ation is equivalent to the reflection in two dimensional space. 




FIG. 6: (Color online) The mean field spinon band structure of 
the ansatz SL[-|, §,7r] and SL[— -j, -1,0] along the k^^ axis when 
Jt=2Je. (a) The first Brillouin zone. Here Q and -Q denote the 
points in which minimum (maximum) of the conduction (valence) 
band of the SL[|-, ^, n] occurs. The K and - K indicate the points 
where the linear band touching between the two bands near the Fermi 
energy of the SL[— -j, ^, 0] occurs, (b) The dispersion of the valence 
and conduction band corresponding to the SL[|-, fj-Tr] ansatz. The 
low energy excitation near the ±Q can be described by the massive 
Dirac particles, (c) The dispersions of the two bands near the Fermi 
level for SL [— -I , § , 0] . The low energy excitation near the ±K point 
can be described by the Dirac particles under the staggered chemical 
potential. 

Cuni form is odd Under both the time-reversal (T) and par- 
ity (Pi) but even under inversion (/). On the other hand 
Cstaggered is odd Under timc-rcversal and inversion but even 
under parity. Here the inversion operation is defined with re- 
spect to the mid-point of the expanded link connecting two 
triangles. In both cases, the combination of the time-reversal, 
parity transformation, and inversion (T • Pi • /) is equivalent 
to the identity operation under which both the Cuni form and 
Cstaggered are invariant,^ 

The SL[-|, -^j tt] is characterized by nonzero Cuniform but 
with vanishing Cstaggered- Therefore it breaks time-reversal 
and parity transformation while it is invariant under the com- 
bined operation. It is a chiral spin liquid state which has a 
finite energy gap. In Fig. |6la) we plot the spinon dispersion 
near the Fermi level corresponding to the valence and con- 
duction bands. The energy gap is minimum at the momentum 
Q=(7r/3,0) and -Q. If we expand the mean field Hamiltonian 
near the dispersion minimum, Q and -Q in the limit of large 
Jt/Je, we can get the following effective low energy Hamil- 
tonian, 

j—^'^Hq){vF[qxToc + QyTy] - mT2}*((7), 

(15) 

where the Fermi velocity vp = JeXel \/3 and the mass m = 
{J^y^^) / {^/iJtXt)- In the above we define the eight compo- 
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nent Dirac fermion field, = {ip\ a cnf2 a a) ™ which 1 
and 2 are the two-component Dirac indices, a and a are in- 
dices for the nodes (±Q) and spins. The Pauli matrix r,y acts 
on the two-component Dirac space. For later convenience we 
define two additional Pauli matrices, jl and a acting on the 
nodal and spin spaces, respectively. 

Since the mass term has the same sign in the two nodal 
positions, integrating out fermions leads to the Chern-Simons 
gauge field action. As a consequence, the charge neutral 
spinon Hall conductivity should be finite. The Chern-Simons 
term stabilizes the spin liquid ground state by providing a 
finite mass to the U(l) gauge field. Therefore the U(l) gauge 
field can only mediate a short range interaction between 
the spinons, which makes the fractionalized particles (the 
spinons) to be the elementary excitations of the spin liquid 
ground state. 

(d) SL[— ^, ^, 0] : A nematic spin liquid state 

The SL[— -I, §,0] is characterized by nonzero C staggered 
but with vanishing C uniform- Therefore it breaks both the 
time-reversal and inversion operation but is invariant under the 
parity transformation. Because the fluxes of the two triangles 
within the unit cell have opposite sign, the six-fold rotational 
symmetry is broken down to the three-fold symmetry. (See 
Fig Hb).) Thus it is a nematic spin liquid. 

The mean field spinon dispersion corresponding to the two 
bands near the Fermi energy is plotted in Fig.|6](b). The spin 
liquid ansatz has a spinon Fermi surface which consists of an 
electron pocket at the K = (27r/3,0) point and a hole pocket at 
the - K point. 

Expanding the mean field Hamiltonian using JeXe/ JtXt as 
an expansion parameter, the following effective low energy 
Hamiltonian can be obtained, 

-^'^\l){vF[qxT^ + qyTy] - Mn,]^{q), 

(16) 

where the fermi velocity vp = JeXe/^Ji and the "staggered" 
field M = (JlxDli^JtXt)- Since the effective chemical 
potentials coming from the "staggered" field M have the op- 
posite signs at the two nodal points, we have both an electron 
pocket (at the K point) and a hole pocket (at the -K point) on 
the fermi surface. 

In contrast to the SL[^,-|,7r] state which has a gapped 
spinon spectrum, the SL[— ^, ^, 0] state has gapless low en- 
ergy excitations. To confirm that the low energy description 
based the above effective Hamiltonian in Eq. (fT6] l is valid af- 
ter including the fluctuation beyond the mean field descrip- 
tion, we have to check whether there are relevant perturba- 
tions which are allowed by symmetry. Especially, some of the 
fermion bilinears, which are made of ^t, can potentially gener- 
ate various mass terms which spoil the low energy description 
ofEq. (O. 

To judge the stability of this spin liquid state, we have to un- 
derstand how the symmetries of the microscopic Hamiltonian 




FIG. 7: (Color online) The generators of the point group symmetry of 
the SL[— -I, ^,0]. Ry maps y to - y while 6*2,1/3 induce the rotation 
by 27r/3 with respect to the center of the dodecagon. 



are realized in the effective continuum theory. Even though 
the original spin Hamiltonian is invariant under the full space 
group transformations, after the gauge theory formulation of 
the problem, the symmetry of the mean field Hamiltonian is 
realized projectively. That is, under the symmetry transfor- 
mation S with the mapping i— >S'(i), the spinon operator /i ,j 
transforms in the following way. 



S :/i,o- — > Gs(i)/s(i),<T, 



where G5 (i) is a phase factor which depends on the symme- 
try operation S, and a local coordinate i. The group of the 
symmetry operations which make the mean field Hamiltonian 
invariant is called the projective symmetry group (PSG)i^2i^ 

To perform the PSG analysis we have to specify the sym- 
metry group of the spin Hamiltonian. The star lattice has the 
point group symmetry generated by the six-fold rotation 
symmetry with respect to the center of a dodecagon and the 
reflections. However, due to the finite fluxes inside triangles, 
the SL[— ^, ^, 0] state breaks some Dq point group symme- 
tries. Especially, the six-fold rotational symmetry is broken 
down to three-fold rotational symmetry. The point group sym- 
metry of the SL[— -I, -I, 0] state is generated by the 27r/3 ro- 
tation (C2,r/3) around the center of the dodecagon and the re- 
flection {Ry) which maps y to - y. The symmetry operations 
which generate the point group of the SL[— ^, ^,0] state are 
depicted in Fig. |7] The SL[— ^, ^, 0] state is also invariant 
under the translations {Tai and Ta2) by the lattice vectors ai 
and 32. In combination with the above point group symme- 
try, the translational symmetry defines the space group of the 
SL[--2-, |,0] state. In addition, the SL[-|, -^,0] state is in- 
variant under the combination (T • /) of the time-reversal (T) 
and inversion (/) as well as the spin rotation. Finally, it has 
the charge conjugation symmetry (C*) via the mapping fia 
— > e,/2-, where Ci = 1 for i=l,2,3 and -1 for i=4,5,6. Under 
these symmetry operations the continuum field transforms 
as follows. 
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(17) 



Using the above transformation rules we can easily check 
that '^^fiz'i' is the only fermion bilinear which is allowed by 
symmetry. Therefore the low energy Hamiltonian in Eq. ( fT6] l 
is valid even after we include the fluctuations and protected 
by the projective symmetry group. Some details about how 
we have determined the transformation rule of the continuum 
fields are explained in the AppendixlB] 

Next we discuss about possible instability of the 
SL[— 7r/2, 7r/2, 0] state. The low energy effective Hamil- 
tonian (Eq. ( fT6l )) which is obtained from the perturbative 
expansion in powers of JeXe/JtXt implies that the electron 
pocket (at the K point) and the hole pocket (at the — K point) 
are nested in the large Jt/Je limit. Therefore the instability in 
the particle-hole channel with the momentum 2K is expected. 
The following two fermion bilinears, rhx='^^T2iJ,x'^ and 
rriy='^^ Tziiy'^ , are especially important in this respect. 
Addition of the mass term Hm = M^rrix + Myfhy to 
the effective Hamiltonian in Eq. (fT6] l leads to the mass 

gap of 2,/m2 



+ M^. Since these mass terms are 

anticommuting with the effective Hamiltonian in Eq. ( fT6l ). 
the pair {rhx,Thy) opens the largest mass gap than any 
other pairs of possible mass terms. Interestingly, {mx,fhy) 
transforms nontrivially under the space group operations. Its 
symmetry property is consistent with some ordered state with 
V3 X VS-type translational symmetry breaking. Using the 
terminology defined in Sec. lIIIBl (Thx,'rhy) transforms as the 
irreducible representation under the enlarged point group 
Gp b. The detailed discussion on the group theory for the star 
lattice is given in Sec. 1111 Bl 

(e) SL[f,f,0] and SL[-f,f,7r] 

We obtain SL[f , f , 0] (SL[-f , f , tt]) phase by putting ad- 
ditional TT flux on every dodecagon on top of SL[^, §,7r] 
(SL[— -I, ^,0]) states. Due to the introduction of the addi- 
tional TT flux, the mean field description requires unit cell dou- 
bhng although the actual physical wave function maintains the 
translational invariance. 

The spinon dispersion of the SL[^, ^, 0] state is described 
in Fig. [Sa). Basically, the structure of the low energy spec- 
trum of SL[^, ^, 0] is similar to that of SL[-|, ^, tt], except 
that the number of the momentum points which support low 
energy excitations is doubled. Both of them are character- 
ized by finite Cu7ii form indicating the time-reversal and parity 
symmetry breaking. Therefore the SL[-|, ^, 0] state is also a 
chiral spin liquid state. The low energy excitations can be de- 
scribed by the effective Hamiltonian similar to Eq. ( fTsT i which 




(b) 



FIG, 
SL 



0.25 J, 



-0.25 J. 



0.25 J. 



0.25 J. 



(Color online) The low energy spinon bands of the 
^ , -1 , 0] and SL[— -I , -I , tt] plotted along the ky=\/3kx direction. 




(a) For SL [ -I , -1 , 0] . The locations of the minimum (maximum) band 
gap are described by rrii (Mi), (b) For SL[— -I, f , tt]. Here mi and 
m3 (m2 and 771,4) indicate the location of the electron (hole) pockets. 



can be obtained following the same procedure we used to de- 
rive Eq. (QSll for SL[f,f ,7r]. 

In Fig.[8jb) we have drawn the low energy spinon excitation 
spectrum of the SL[— -I, ^, tt] state. There are two electron 
pockets (around mi and ma) and two hole pockets (around m2 
and m4). It is characterized by finite C staggered showing bro- 
ken time reversal and inversion symmetry. Since the fluxes in- 
side up-triangles and down-triangles have opposite signs, the 
six-fold rotational symmetry is broken down to three-fold ro- 
tational symmetry. Therefore it is another nematic spin liquid 
state. 



III. DIMER PHASES 
A. Large-N approach 

According to the pioneering work by D.S.Rokhsar,"*" when 
the lattice system is dimerizable, the best mean field ansatz 
is one of dimerized states in the large-N limit of the SU(N)- 
generalized Heisenberg model. Here we call a lattice to be 
dimerizable when it is possible to make every site belong to 
a dimer and a lattice site be paired with one and only one of 
its neighboring site. In particular, when every dimer is lying 
on the link which has the maximum exchange coupling Jmax, 
the dimer state belongs to the ground state manifold of the 
mean field Hamiltonian. In terms of the variable Xij^ we have 
finite only on the dimers lying on the link which has the 
maximum spin coupling Jmax ■ 

The star lattice is dimerizable with respect to Jg. Therefore 
when Je is larger than Jt, it has a unique dimerized ground 
state (we call it the Jg-dimer VBS) in which every dimer is 
lying on an expanded link connecting neighboring triangles. 
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FIG. 9: (Color online) (a) The Je-dimer VBS state. This is the 
ground state in the large-N limit when Je > Jt. 

(a) (b) A 

^ o 

FIG. 10: (Color online) (a) The representative local dimer config- 
uration when Jt > Je. A dimer lying on an expanded link con- 
nects two neighboring filled triangles which are supporting dimers 
on them, (b) The 18-site flippable loop which consists of the alter- 
nating bright(blue) and dark(black) thick lines. 

In Fig. |9] we describe the geometric arrangement of singlet 
dimers of the Je-dimer VBS phase. 

On the other hand, the Rokhsar's general theorem cannot 
be applied when Jt is larger than Je- This is because the star 
lattice is not dimerizable with respect to the Jt links and every 
dimer configuration defined on the star lattice contains a finite 
number of dimers lying on the J^ links. Therefore it is possi- 
ble that the translationally invariant mean field ansatz can be 
the ground state even in the large-N limit. 

When Jt > Je, we have to maximize the number of the 
dimers lying on triangular links to minimize the ground state 
energy of dimerized states. Since every triangle can support 
a single dimer at most (we call the triangle with a dimer ly- 
ing on it a filled triangle), the remaining unpaired lattice point 
of the filled triangle has to be a part of the dimer lying on 
an expanded link. In other wards, every dimer lying on an 
expanded link is connecting two filled triangles and this de- 
scribes a representative local dimer configuration of the low- 
est energy dimerized states when Jt > Je- (See Fig. [TOf a).) 
Using this local dimer configuration as a building block we 
can construct infinite number of degenerate dimerized ground 
states. 

To understand how the degeneracy of these dimerized 
phases is lifted by fluctuations, we consider the 1/N correc- 
tions to the ground state energy. In Ref.[l^ Read and Sachdev 
investigated the 1/N correction systematically for a similar 
problem defined on the square lattice. We first review the 
main ideas of their work briefly and extend the approach to 
our star lattice problem. For a given dimer configuration, Xij 
we include the fluctuations 6xij as Xij = Xij + ^Xij- Here 
Xij is nonzero only on the link supporting a dimer lying on 
it. Expanding the effective action to the quadratic order in 



the fluctuations, the ground state degeneracy of the dimerized 
states on the square lattice could be lifted by the following 
terms, SSef / oc N Xij^XjkXkiSxu- Here i,j, k, and I indicate 
the four corners of a square plaquette. When a pair of links 
lying in parallel are occupied by two dimers (xij and xu) 
and the remaining pair of the links are assigned to the fluc- 
tuations (Sxjk and Sxh), the SSeff term above can induce 
additional lowering of the ground state energy. It means that 
dimer configurations which support the maximum number of 
the parallel dimer pairs span the ground state manifold. The 
four-fold degenerate columnar valence bond solid is selected 
as the ground state following these procedures. 

The above idea of the 1/N-correction can also be rephrased 
in the following way. For every square plaquette composed 
of two parallel dimers, we can define a loop which consists 
of alternating occupied and empty links. Here we call such a 
loop as a flippable loop''' because two degenerate dimer con- 
figurations are connected via a loop flip, i.e., the interchange 
of the occupied and empty links. The 1/N-correction captures 
the energy lowering through the resonance process which can 
also be described as a loop flip. The resulting ground state (a 
columnar dimer state) supports the maximum number of the 
flippable loops. This idea can be generalized to the higher 
order corrections and the degeneracy of dimerized states be- 
gins to be lifted from the lowest order correction correspond- 
ing to the smallest flippable loop. Marston and Zeng^*' dis- 
cussed the effect of the 1/N-correction on the degeneracy lift- 
ing process for the kagome lattice antiferromagnet. There, 
the first term that lifts the degeneracy involves the six-site 
flippable loop, which is the so-called perfect hexagon with 
three dimers on it. The valence bond solid ground state of 
the kagome lattice, which contains 36-site within the unit cell, 
results from the condition of maximizing the number of the 
perfect hexagons i'^''^'^" The similar idea was also applied to 
the square-kagome antiferromagnet.^' 

In the star lattice problem with Jt > Je, the dimerized 
ground states are constructed by repeating the representative 
local dimer configuration displayed in Fig. [TOf a). In this 
ground state manifold, the smallest flippable loop contains 
18 sites with a dodecagon at the center, which is shown in 
Fig.[TOfb). Here when the bright (blue) thick link is occupied 
by a dimer, the neighboring dark (black) thick link is empty 
and vice versa. By interchanging the roles played by the bright 
(blue) links and the dark (black) links, two degenerate dimer- 
ized phases can be connected. 

Therefore the fluctuation corrections pick the patterns that 
maximize the number of the 18-site dimer units participating 
in the 18-site flippable loops. In Fig. [TTT a) we show the va- 
lence bond solid order which has the maximum number of 
the 18-site flippable loops. Among the six neighboring do- 
decagons around an 18-site dimer unit, three can be the centers 
of the 18-site dimer units. This is in contrast to the case of the 
kagome lattice problem. There, none of the six neighboring 
hexagons around a perfect hexagon can be perfect hexagons. 
In fact, the valence bond solid states on the star lattice have 
similarity with those on the square lattice. In the case of 
the square lattice, among the four neighboring square plaque- 
ttes around a central plaquette supporting two parallel dimers. 
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FIG. 11: (Color online) Two low energy valence bond solid (VBS) 
order which are coming from the 1/N correction, (a) The columnar 
18-site VBS. (b) The box 18-site VBS. 



FIG. 12: (Color online) (a) The intersections between the reflection 
planes and the lattice plane, (b) The lattice vectors bi and b2 corre- 
sponding to the y/S X y/S ordered states. 



half of them (two square plaquettes) can support two parallel 
dimers. By maximizing the number of the square plaquette 
composed of two parallel dimers, the columnar valence bond 
solid emerges. 

Based on the similarity with the square lattice problem, we 
can call the valence bond solid in Fig. [TTT a) as a columnar 
18-site valence bond solid. In this figure all the dodecagons 
except the central one support the 18-site dimer unit. The 
three-fold degeneracy of the columnar 18-site VBS comes 
from the broken translation symmetry. In addition we also 
consider another low energy valence bond order which is dis- 
played in Fig. [TTT b). Here the 18 links around the 18-site 
unit have the same finite value of Xij- In analogy with the 
square lattice problem this phase can be called as the 18-site 
box VBS phase. The translational symmetry breaking results 
in the three-fold degeneracy in this phase as well. In the recent 
numerical study by G. Misguich et al.^'', this phase was sug- 
gested as a possible valence bond solid ground state when Jt 
> 1.3 Je- We expect that the 1/N corrections would select the 
18-site columnar VBS as the ground state that can maximize 
the resonance energy gain from the smallest flippable loop. 
However, since the length of the flippable loop is quite large 
compared to those on the square and kagome lattice problem, 
the energy difference of the two candidate valence bond solids 
in Fig.[TT|couId be very small. 



a dodecagon and E is the identity operator. Ra indicates 
the reflection with respect to a plane orthogonal to the lattice 
plane. The reflection planes (denoted by a) are described in 
theFig.fni'a). 

The space group, Gs, of the star lattice is generated by 
the translation group, Gt, and the Dq point group. An ele- 
ment of the space group can be written using the Seitz oper- 
ator {gDg|t}, where gu^ is an element of the Dg group and 
t=niaiH-n2a2. (n,i and n2 are integers.) The action of a Seitz 
operator on a lattice point r is defined as {goe |t}r — toe^'+t- 
Note that the translation group, Gt, is an invariant subgroup 
of the space group Gs and the point group is the coiTe- 
sponding factor group, i.e., De= GsIGt- 

To understand the symmetry of the enlarged unit cell, we 
define another translation group, Gr.b, whose elements can 
be written as. 



GT,b ^{{E\nihi + nsba}; m, ^2 S Z} (18) 



B. Group theoretical approach to \/Z x ^/i bond orders 

The columnar and box 18-site VBS phases discussed in 
the above section break the lattice translational symmetry and 
are described by the enlarged VS x Vs unit cell. Here we 
perform the detailed symmetry analysis on the bond order 
that are compatible with the -\/3 x -\/3 enlarged unit cell. 

(a) Group theory for the star lattice 



The star lattice has the point group symmetry. The 
twelve elements of the Dq group are as follows. 

Dq —{E, Cq, Gq, Gq, Cq, CQ,Ra,Rb, Rci Ra, Rb, Rc}, 
where Gq means the rotation by 27r/6 around the center of 



where bi = ai + a2 and b2 = 2ai — a2 are the lattice vectors 
corresponding to the \/3 x a/3 ordered state. (See Fig.[T2tb)) 
Since Gx^b is an invariant subgroup of the space group (^5, the 
enlarged point group G p b can be defined as the factor group 
Gp,b= GslGT,h- Therefore the elements of the space group 
can be written as {gcpb I "ibi + n2\i2] in which gopt, is an 
element of the enlarged point group, Gp^y,. (Note that similar 
approach was used by Hermele et aL^S to investigate the sym- 
metry properties of the object invariant under the translations 
by 2ai and 2a2 on the kagome lattice.) 

The construction of the Gp^b group is straightforward, 
whose elements can be written using the Seitz operator 
{(7£)g|t} with t=0, ai, a2. The 36 elements of the {gzjglt} 
can be grouped into the nine conjugate classes. 
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TABLE I: The Character table of the enlarged point group G p,b 



Ce ={{i?|0}}, 

CT={{E\a,},{E\a2}}, 

Cl={{Cl\0},{Cl\0}}, 
CsT ^{{Ce\0}, {Cim, {Celai}, {C||ai}, {CM, {C||a2}}, 
C|t ={{C||ai}, {C||ai}, {CHa^}, {C^^la^}}, 
Clr^{{Cl\0},{ClM,{C!\a2}}, 

Rl ={{i?a|0},{i?fc|0},{i?c|0}}, 

i?iT ={{i?a|ai}, {Rb\ai}, {Rc\ai}, {RaM, {RbM, {RM} 
i?2T ^{{Ra\0}, {Rb\0}, {Rem, {RaM, {RbM, {RcM, 
{RA\a2},{RB\a2},{Rc\a2}}- 



FIG. 13: (Color online) Description of the pair of the £3 bond or- 
dered states which form a basis of the Es irreducible representa- 
tion, (a) Ci=\/3 for the thick light (red) links, -\/3 for the thick dark 
(black) links, and zero for the thin solid links, (b) c;=l for the thick 
light (red) links, -2 for the thick dark (black) links, and zero for the 
thin solid links. 




Table U displays the character table corresponding to 
the enlarged point group Gpb. It consists of the four one 
dimensional irreducible representations {Ai, A2, Bi, and B2) 
and the four two dimensional irreducible representations, 
(a=l,2,3, and 4) and a four dimensional representation Q. 
Note that the four one-dimensional irreducible representa- 
tions and the two dimensional representations Ei and E2 are 
simple extensions of the six irreducible representations of the 
original Dq point group. They describe states which are in- 
variant under the lattice translations by ai or a2 as is reflected 
in the column for Ct in Table. U Therefore the \/3 x \/3 type 
orderings can be described only through the remaining three 
irreducible representations £^3, E4 and Q. It turns out that 
the two dimensional irreducible representations E-s and E4 
are especially important considering the consistency with the 
numerical resulti^ 

(b) X bond ordering patterns 

Here we focus on all possible bond ordering patterns which 
are compatible with the \/3 x %/3 enlarged unit cell. Labeling 
the 27 links inside the VS x \/3 unit cell by | Z) , a bond ordering 
pattern is given by the linear combination of the \l) as, 



FIG. 14: (Color online) Description of the pair of the E^ bond or- 
dered states which form a basis of the E3 irreducible representa- 
tion, (a) c;=\/3 for the thick light (red) links, -\/3 for the thick dark 
(black) links, and zero for the thin solid links, (b) c;=l for the thick 
light (red) links, -2 for the thick dark (black) links, and zero for the 
thin solid links. 



the enlarged point group Gp.b whose decomposition into the 
irreducible representations is given by 



bond 



= 2Ai ®B2®Ei® 2E2 ® 2E3 ®E4® 3Q. 



Notice that Fbond supports two independent E^ irreducible 
representations (we call them as E^ and E^ , respectively) 
and one E4 irreducible representation. The bond ordering pat- 
terns which constitute a basis of each iiTeducible representa- 
tion are displayed in Fig. [13] Fig[T4l and Fig. [TSl describing 



E^ , and E4 irreducible representations, respectively. 



A bond order transforming as an i?3 irreducible representa- 
tion can be represented by a linear combination of states Uke 



|i;3bondorder) ^ai\E^{a)) + a2\E^ib)) 

+ a3\Eiia))+a4\Ei{b)), 



(20) 



I bond order) 



(19) 



in which \E^ 



where c; is proportional to the amplitude of the singlet corre- 
lation on the link I, i.e., qcxx/. The vector space spanned by 
\l) constitutes a reducible representation (defined as Fbond) of 



('^)) = '^'10 with Cl specified in Fig. flJl a). 
The other three basis states \E^ib)), (a)), and (fe)) 
are defined following the same way. 

Interestingly the bond ordering patterns of the two valence 
bond solid states, the columnar and box 18-site VBS, are given 
by the following superposition of states. 
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FIG. 15: (Color online) Description of the pair of the E4 bond or- 
dered states which form a basis of the £4 irreducible representa- 
tion, (a) ci=V3 for the thick light (red) links, -\/3 for the thick dark 
(black) links, and zero for the thin solid links, (b) In each triangle 
the link belonging to the central dodecagon has the ci=2 (-2) if it has 
red (black) color. The other two links of the triangle have c;=l (-1) 
if they have red (black) colors. 



iColumnarVBS) cx |uniform) - \E^{b)) - \E^{b)), 



Box VBS) cx luniform) + -\E^{b)) + -\Ei{b)), (21) 



where juniform) = J2i 10- Since both the columnar and box 
18-site VBS are invariant under the reflections Ra, Rb, and 
Rc, \E^{a)) and \E^{a)) have no contribution. Superposi- 
tions of the \Eiib)) and \E^{b)) can induce more general 
bond ordering patterns other than those described in Fig. [TT| 
Finally, since E4 irreducible representation always breaks the 
reflections Ra, Rb, and R^ (See Fig. [15), we neglect bond 
orders transforming as E4 irreducible representation. 





Emf (unprojected) 


Emf (projected) 


Dimer 


-0.625 


-0.625 


SL[0,0,0] 


-0.498 


-0.647 


CT [ TT IT 1 

J)L[-2 , -2 ,7rJ 


-0.553 


-0.624 


SL[-f,f,0] 


-0.553 


-0.616 


SL[0,0,7r] 


-0.498 


-0.654 


SL[f,f,0] 


-0.552 


-0.617 


SL[-f,f,^] 


-0.552 


-0.614 



TABLE II: The ground state energies of the various mean field ansatz 
when Jt=2Je. The energies are measured in unit of Je 



B. Projected wave function study 

In the above mean field calculation, the single occupancy 
constraint is imposed only on average. Therefore the mean 
field wave functions contain unphysical states with zero or 
two fermions at a point. To obtain physical spin wave func- 
tions we therefore perform a numerical Gutzwiller projection 
on the mean field wave functions. The ground state energies 
of the projected states are computed numerically using the 
variational Monte Carlo (VMC) method'*'-'*^. The resulting 
energies are displayed in Fig[T6]for a range of Jt/Je where 
we have optimized the state with respect to Xt/Xe for each 
value of Jt/Je- Table HIl shows the numerical energy val- 
ues for Jt/Je=2.Q to facilitate a comparison with the mean 
field numerics. We find that Gutzwiller projection dramati- 
cally changes the relative ordering of the various states, and 
that the state SL[0, 0, tt] appears, upon projection, to be the 
lowest energy spin liquid over the entire parameter range. 



C. Bond operator approach 



IV. GROUND STATE ENERGY 

In this section we compare the ground state energies of var- 
ious spin liquid states. 



A. Mean field tlieory 

The ground state energies of various spin liquid states for 
Jt=2Je are shown in Tablellll In addition to the translationally 
invariant spin liquid states with finite Je and Jt, we have also 
considered a decoupled dimer phase for comparison. A de- 
coupled dimer phase, which has the lowest mean field ground 
state energy for Jt > Je, can be built based on the local dimer 
configuration described in Fig. [TOta). The columnar 18-site 
valence bond solid (VBS) displayed in Fig. fTTT a) is an ex- 
ample. According to the mean field calculation, the dimer 
state has lower ground state energy than any other translation- 
ally invariant spin liquids. Among the spin liquid phases with 
translational invariance, the four ansatz having the 7r/2 flux 
inside triangles have lower energies than those having zero 
flux inside triangles, i.e., SL[0, 0, 0] and SL[0, 0, tt]. 



According to the projected wave function study, SL[0,0,7r] 
state is the ground state over a wide parameter space. How- 
ever, the SL[0,0,7r] state is unstable due to the confinement 
in the 2h-1 dimensional pure gauge theory. It is also incon- 
sistent with our expectation for the Jg ^ Jt limit. When 
Je 3> Jt, the Je-dimer VBS phase (See Fig. |9l) is the ex- 
act ground state. In addition the recent exact diagonaliza- 
tion study shows that the Jg-dimer VBS phase remains as the 
ground state up to the isotropic limit of Je = Jt-^^ Therefore 
SL[0,0,7r] state should have higher energy than Jg-dimer VBS 
phase at least in some finite range of < Jt/Je < 1- This 
discrepancy comes from the lack of the interdimer interaction 
in the decoupled dimer limit. For the description of dimer- 
ized phases beyond the decoupled dimer limit, we undertake 
the self-consistent bond operator approach. If the cor- 
rection coming from the inter-dimer interaction is significant, 
we also have to check the possibility that the true ground state 
is a valence bond solid even when Jt > Je- 

In the bond operator formulation, the dimer singlet degrees 
of freedom are used as natural building blocks and the quan- 
tum corrections coming from the triplet fluctuations can sys- 
tematically be investigated."*'''"^^ Here we present a brief expla- 
nation of the bond operator formulation. Let us consider the 
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FIG. 16: (Color online) The ground state energy per spin of the var- 
ious spin liquid ansatzes computed using Gutzwiller projection, (a) 
Energies for < Jt / Je < 1 measured relative to the Je-dimer VBS 
state shown in Fig|9]which has an energy — 3Je/8 per spin, (b) En- 
ergies for 1 < Ji/Je < 3 measured relative to the -\/3 x \/3 VBS 
state shown in Fig[TT]which has an energy — Jt/4 — Je/8 per spin. 
Note that SL[0,0,7r] (solid line is a guide to the eyes) has the lowest 
energy over the whole parameter space. SL[0,0,0] has the second 
lowest energy for a wide range for Jt > Je- For Jt = 2Je = 2, the 
energy per spin of all six spin liquid ansatzes are given in Tablellll 
(These computations were carried out on a system with 6x6 unit 
cells, i.e., with 216 spins, and the statistical error bars on the energy 
are of the order of the symbol size). 



two •5=1 spins constituting a dimer singlet, Sji and S^. 
The Hilbert space is spanned by four states that can be taken 
as a singlet state, |s), and three triplet states, jij.), \ty) and 
\tz). Then, the singlet and triplet boson operators are intro- 
duced such that each of the above states can be created from 



V2 



(I n)-UT)), 



IM=4|0) = -^(| TT)-|U)), 
|t,)=4|0) = -^(|TT) + IU)), 

|i.>-4lo> = ^(in> + UT)). 

To eliminate unphysical states from the enlarged Hilbert 
space, the following constraint needs to be imposed on the 
bond-particle Hilbert space: 

where a = x,y, and z, and we adopt the summation con- 
vention for the repeated indices hereafter unless mentioned 
otherwise. 

Constrained by this equation, the exact expressions for the 
spin operators can be written in terms of the bond operators: 



1 



where SaPj is the third-rank totally antisymmetric tensor with 

£xyz — 1- 

Utilizing the bond operator representation of spin operators, 
the Heisenberg spin Hamiltonian in Eq. ^ can be rewritten 
solely in terms of bond particle operators. Since all dimers 
of Je -dimer VBS phase are symmetry equivalent, the singlet 
condensate density {s\) and the chemical potential ii\ can be 
set to be (sj) = s and /ii=/x in our mean field theory. Here i 
denotes the location of dimers. The hard-core constraint on 
the bond-particle operators is imposed by adding the follow- 
ing Lagrange multiplier term, i?^ ~ ~ X^i /^(^^ + ^a^i" ~ 
The resuling Hamiltonian can be written as follows: 



H = Ne 







Quad 



Quartic 5 



(22) 



where 

Hquad 



Je 



JtS^ 



^{4ijo+tiatja+H.C.}, (23) 



and 



(iJ> 



-^Quartic — . ^ ^^ ^al3'^^a.^.y^\Q^\-yt\^^^\y • 
(iJ> 

In the above, N is the number of unit cells and 

3 



£0 = 3 



(24) 



(25) 
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Jt/J. 

FIG. 17: (Color online) Comparison of the ground state energy of 
SL[0,0,7r]tothatof Je-dimerVBSforO< Jt/Je < 1. To emphasize 
the importance of the interdimer interactions we present the energies 
of Je-dimer VBS obtained in three different ways. The 'Decoupled' 
indicates the energy of the decoupled dimers. 'Quartic' ('Quadratic') 
is the energy from the bond operator theory with (without) the quartic 
interaction effect. 

The quartic interactions between triplet particles are decou- 
pled using the mean field order parameters P and Q, where 
P = (ij^ija) and Q = {tiatja}- Here P and Q denote the 
diagonal and off-diagonal triplet correlations between neigh- 
boring dimers. These two order parameters P and Q together 
with s and /i are determined self-consistently by solving the 
coupled saddle point equations4i^ 

The ground state energy of Jg-dimer VBS phase obtained 
from the self-consistent bond operator calculation is displayed 
and compared to the energy of SL[0,0,7r] state in Fig.[T7] Here 
we have obtained the energy of Je-dimer VBS phase in three 
different ways. If we neglect the inter-dimer couplings (the 
decoupled dimer limit) completely, the energy is independent 
of Jt/Je ■ The inclusion of the inter-dimer interaction lowers 
the ground state energy significantly. In the end, -dimer 
VBS phase has the lower ground energy than the SL[0,0,7r] 
state over the entire parameter range of 0< Jt/Je <1 when we 
include the quartic interactions. The inter-dimer interactions 
generate huge correction to the ground state energy of dimer- 
ized phases. 

Now we concentrate on the other limit where Jt>Je- In 
contrast to the J^^Jt limit, it is nontrivial to identify the 
ground state even when we restrict our attention to valence 
bond solid phases. Taking into account the information from 
the exact diagonalization study and 1/N fluctuation from the 
large-N limit, we suggest the columnar 18-site VBS phase as 
a promising candidate for the ground state as explained below. 

We apply the bond operator approach to the columnar 18- 
site VBS phase. The nine dimers within the unit cell can 
be divided into two groups. One group is made of the six 
dimers lying on the triangular links. Note that all these six 
dimers are lying on a dodecagon. (See the central dodecagon 
in Fig. [TTT a).) We call such a dodecagon surrounded by six 
dimers as a "perfect" dodecagon. The remaining three dimers 
lying on the expanded links make the other group. Every 
dimer belonging to the same group is symmetry equivalent 
as one can easily notice from the patterns around the central 
"perfect" dodecagon in Fig. [TTT a). To apply the bond operator 




1J) 2J> 3.0 4.0 

Jt/J. 



FIG. 18: (Color online) Comparison of the ground state energies of 
the columnar 18-site VBS to that of SL[0,0,7r] state, (a) Energetics 
for Jt>Je. The energies of the columnar 18-site VBS are obtained 
in three different ways as in Fig[T7] (b) The relative energies ob- 
tained by subtracting the decoupled dimer energy. Note that there is 
a level crossing around Jt/Je«2.4 between the energy of SL[0,0,7r] 
state and that of the columnar 18-site VBS including the quartic in- 
teraction. 



approach we have to introduce two independent sets of order 
parameters to distinguish the two different groups of dimers. 
We use Se and (st and /if) to indicate the singlet condensate 
density and the chemical potential corresponding to the ex- 
panded (triangular) link. To decouple the quartic triplet inter- 
actions we introduce two sets of the order parameters, that is, 
{Ppp, Qpp} and {Pep, Qep}- Ppp (Qpp) describes the diago- 
nal (off-diagonal) correlation between the neighboring dimers 
lying on a "perfect" dodecagon. On the other hand Pep (Qep) 
describes the diagonal (off-diagonal) correlation between a 
dimer lying on an expanded link and its neighboring dimer ly- 
ing on a "perfect" dodecagon. We have determined the eight 
parameters Sa, /ia (a = t or e), and Pp, Qp {P = pp or ep) self- 
consistently by solving the coupled saddle point equations. 

The self-consistent solution shows that Pep=Qep=0, Sg^l 
and /ie=-3/4 Je. Since Pep and Qep describe the coupling be- 
tween the dimers lying on expanded links and the dimers ly- 
ing on "perfect" dodecagons, these two groups of dimers are 
completely decoupled when Pep=Qep=0. In this situation, ev- 
ery dimer lying on expanded links is decoupled from the sur- 
rounding, leading to Se=l and /ie=-3/4Je. The triplet fluctu- 
ations are confined inside every isolated perfect dodecagon, 
which is reflected in the finite Ppp and Qpp values. This in- 
teresting structure would result in the highly localized triplet 
excitation spectrum. 
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The ground state energies of the columnar 18-site VBS and 
SL[0,0,7r] are compared in Fig. [18] The energies of the colum- 
nar 18-site VBS are obtained in three different ways again. 
That is, for the decoupled dimer limit, including the inter- 
dimer coupling neglecting quartic interactions, and finally in- 
cluding the inter-dimer quartic interactions. For clarity we 
also calculated the energy difference relative to the decou- 
pled dimer energy as shown in Fig. fTSl' b). Interestingly, there 
is a critical ratio {Jt/ Je)c~ 2.4 beyond which the colum- 
nar 18-site VBS becomes the ground state when we include 
the quartic triplet interactions. Even though the critical ra- 
tio {Jt/ Je)c is a bit larger than the suggested phase bound- 
ary from the numerical study^'', the existence of the critical 
values of {Jt/Je)c is quite encouraging. In particular, be- 
cause the slopes of the lines in Fig. [Tsl a) are almost parallel, 
small additional energy correction could induce a large shift 
of the crossing point as shown in Fig. [TSl b). Since the sim- 
ple Hartree-Fock approximation does not take into account the 
fluctuations coming from the cooperative interaction between 
the dimers on the expanded links and those on the perfect do- 
decagons, we expect that the quantum correction beyond the 
Hartree-Fock limit could shift the energy level crossing point 
down to ( Jf/ Je)c~1.3 as suggested by the numerical study 



V. INSTABILITY OF SL[0,0,7r] SPIN LIQUID AND 
VALENCE BOND SOLIDS 

Summarizing the previous discussions, Jg-dimer VBS 
phase is the ground state for Jt/Je < {Jt/Je)ci while the 
columnar 18-site VBS is the ground state in the opposite limit 
of Jt/Je > {Jt/Je)c2. (Here {Jt/Je)ci < {Jt/Je)c2.) Al- 
though this result is obtained based on the energy compari- 
son with SL[0,0,7r] state, one may still expect that the two va- 
lence bond solid states are intimately related to the SL[0,0,7r] 
state. In particular, the Jg-dimer VBS and the columnar 18- 
site VBS may arise as a consequence of the confinement in 
the SL[0,0,7r] spin liquid state. In this section we describe the 
possible relation between these two valence bond solid phases 
and the SL[0,0,7r] state. 



A. Spinon confinement and uniform bond orders 

Due to the finite spinon gap, the U(l) gauge field is the 
only low energy excitation in the SL[0,0,7r] state in the long 
wavelength limit. Since the compact U(l) gauge theory with- 
out matter field is confining in 2+1 dimension, we expect that 
the monopole proliferation would lead the SL[0, 0, tt] ansatz 
to some confined phases. To understand the properties of 
the confined phases resulting from the monopole condensa- 
tion, we have to determine the symmetry properties of the 
monopole operators. 

Here we discuss the possibility that the monopole oper- 
ators are invariant under all possible symmetry transforma- 
tions. The Je-dimer VBS phase, which is the ground state for 
Jt/Je < 1, is invariant under space group operations. The 
bond ordering pattern of the Jg-dimer VBS phase belongs to 




FIG. 19: (Color online) Two bond ordering patterns transforming as 
the Ai irreducible representation. The thick solid (red) link has c; = 
1 while the thin solid link has c;=0. (a) Je-bond order, (b) Jt-bond 



the trivial Ai irreducible representation of the point group. 
Therefore if we interpret the Jg-dimer VBS phase to be in- 
duced by the confinement transition, which is reasonable in 
the limit of Jt/Je ^ 1, this reflects the fact that monopole 
operators are invariant under symmetry transformations. 

Extending the group theory analysis we performed in 
Sec. 1111 B1 we investigate all possible bond orders invariant un- 
der the space group operations. These are displayed in Fig. [19] 
Here we have finite singlet correlation (c/ ^ 0) only on the 
thick solid (red) links. The bond order in Fig.[T9la) is nothing 
but the Je -dimer VBS phase. On the other hand, the bond or- 
der in Fig.[T9fb) has finite c/ only on the triangular links (we 
call it as a Jt-bond ordered phase). We expect the Jt-bond 
ordered phase is the natural low energy bond ordering pattern 
when Jt^Je- Since the arbitrary superposition of these two 
orders follows the same Ai UTeducible representation, we ex- 
pect the actual ground states would have finite ci values over 
all the links on the lattice. However, it is natural to expect 
that the c; on the expanded (triangular) link would be larger 
than that on the triangular (expanded) link when Je > Jt (Je 
< Jt)- Therefore the bond ordered phase corresponding to the 
Ai irreducible representation successfully describes the low 
energy manifold over the whole parameter range of Jt/Je- In- 
terestingly, the recent work by Choy and Kim**** has suggested 
that the same bond ordered states are the ground states of the 
same model Hamiltonian in the strong quantum limit based on 
the bosonic Sp(N) approach. 

Since the spinon bandstructure of the SL[0,0,7r] state does 
not change qualitatively by varying Jt/Je, we expect that the 
change of Jt/Je ratio would not affect the trivial monopole 
quantum number Therefore if the ordered phase is coming 
from the confinement transition, it will transform trivially un- 
der the symmetry operations. However, it is also possible that 
the instability of the SL[0,0,7r] state is caused by the inter- 
actions between spinons leading to some broken symmetry 
phases. We discuss about this possibiUty in the following sec- 
tion. 



16 



B. Instability induced by interactions between spinons 

Here we investigate the instability of the SL[0,0,7r] state 
coming from the interactions between spinons and the sym- 
metry properties of the resulting ordered phase. Especially we 
focus on the instability towards the states with the \/3 x 
type translational symmetry breaking. As shown in Fig. |4] 
the valence band is completely flat without any preferred mo- 
mentum. However, the conduction band supports several dis- 
persion minima. The four minimum points of the conduction 
band are given by 



mg =-m2 = (-,0) 
6 



and m, = -m, = (^,^). 



Interestingly if we double the vectors connecting neighboring 
minimum points, they sit on the Brillouin zone corners which 
are nothing but the momentum corresponding to the a/3 x V3 
ordering. Motivated by this observation we study the symme- 
try properties of the bound states made of low energy fermions 
near the conduction band minima. 

We introduce the fermion fields 'I'i which describe the low 
energy excitations near the four conduction band minima m; 
(i= 1,2,3 and 4), 
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n=l 



'(f^«)n/«(x), 



where Vi is the eigenvector of the mean field Hamiltonian 
at the momentum mi, and /„ is a slowly varying fermion 
field near the conduction band minimum, with n labelling the 
twelve sites within the unit cell. To determine the symme- 
try of bound states made of the above low energy fermions, 
we have to understand how the symmetries of the microscopic 
Hamiltonian are realized in the effective continuum fields, ^t^. 
Here we follow the same procedure which we use to deter- 
mine the transformation properties of the continuum field for 
the SL[— ^, -1,0] state in Sec. IIIBI To extract the necessary 
information on the transformations of the continuum field, we 
consider a finite system of a 6x 12 unit cell. By solving the 
mean field Hamiltonian on this finite system, we determine the 
properties of the eigenvectors at the four momentum points, 
m, . The detailed explanation of the procedure as to how to 
determine the symmetry of the continuum fields is discussed 
in the Appendix iB] 

Through the projective symmetry group analysis, we de- 
termine the following transformation properties of the contin- 
uum fields. 




/o 


e * 3 











1 
























1 




*2 







e' 3" 






U4/ 



Ta 



e B 








e*'6 











) 










1 




V 



V2 








— i 
V2 



a/2 
1 
a/2 








1 

-e ' 3 
a/2 

-1 
V2 , 
ie 
a/2 



^ 






a/2 
-1 
a/2 






a/2 
-1 
a/2 

-e'f 
V2 

i 





^3 
^4 



/ ^-1 

^3 

V *4 



We first investigate the symmetry of all possible fermion 
bilinears which can be written as j. For the descrip- 

tion of the 4 by 4 unitary matrix M^j we introduce two sets of 
the pauli matrices and fij. Here is acting on the space 
spanned by either (^'i,^'3) or (^'2,5'4). On the other hand 
Hi is defined in the space spanned by ('^1,^2) or (^'3,^'4). 
Among the sixteen possible bilinears, there are only two terms 
which can form a basis of the enlarged point group Gpy,. 
These are 'flra/io'I'j and '^\to^q'^ j which transform as the 
A2 and Ai irreducible representation of the enlarged point 
group Gp^Yt, respectively. However, since these two bilinears 
have zero total momentum, they cannot describe the \/3 x a/3 
type symmetry breaking. The net momentums carried by the 
other fourteen bilinears are neither zero nor K=(^,0). There- 
fore the transformation properties of them are not compatible 
with the symmetry of the a/3 x a/3 type enlarged unit cell. 

Next we consider the instability in the particle-particle 
channels. Defining the pairing amplitude as /S.ij='^i^ j, we 
have sixteen different A, j . We first omit the indices for the 
spin degrees of freedom and study how they transform un- 
der the space group symmetry operations. It can easily be 
checked that the sixteen pairing amplitudes are divided into 
the four different sets Hi, 112, ^^i and ^2 which transform 
independently. The six-component vectors Hi and the two- 
component vectors 17,; are given by 



Hi 



n 



2 — 



Al3 

A24 



/e-*3An +e'3A33 \ 
e^^tAii - e't A33 

Ai3 + A31 

e~*t A22 + e"'? A44 
e^*t A22 - e'f A44 
A24 + A42 

^e-^tAi2 + e^f A34 \ 
e"*t A12 - e'f A34 

Ai4 — A32 

e-*tA2i +e'^A43 
e~*t A21 - e'^ A43 
A41 - A23 



A31 
A42 



,f^2 = 



Al4 

A41 



A32 
A23 
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Under a space group symmetry operation S, they transform in 
the following way, 

n, ^As(n,)n, and n,~^Bs{n,)n„ 

where 745(11^) (Bs{^i)) is the 6 by 6 (2 by 2) matrix repre- 
senting the symmetry operation S, whose detailed expressions 
are displayed in the AppendixICl 

Since the pairing amplitude is not a gauge invariant 
object, we have to look into the symmetry of the bilinears such 
as Dij^ki=^ij^ki- Even though the number of all possible 
tensors Dij ^i is very large, we can reduce the complexity of 
the symmetry analysis by focusing on the objects carrying the 
momentum compatible with the \/3 x \/3 ordering. This idea 
leads us to exclude Dij ki made of the basis 112 and $12- In 
addition. Hi and fti have opposite spin parities, that is. Hi is 
spin singlet while J7i is spin triplet. Therefore all we have to 
consider are the terms like nlMnlli and r2|Msiili. 

First, we define a set of pauli matrices acting on the space 
spanned by fii. Using the transformation properties of i7i 
under the space group, we obtain a pair (f2|ra;ili,fi|Tj^i7i) 
which has the momentum K=(^,0) and transforms as the 
irreducible representation of the enlarged point group, Gp.b. 

To understand the symmetry of nlMnlli, we introduce a 
set of the Gell-Mann matrices Aa (a=l,....,8)'*'' as well as the 
Pauli matrices r^. For convenience, we also define the ma- 
trix, A9=y^l3 where I3 is the 3x3 identity matrix. The 
Gell-Mann matrices are acting on the space spanned by ei- 
ther (Hi 1, Hi 2, Hi 3) or (Hi 4, Hi 5, Hi e) and connects 
these two three-component vectors. Here Hi „ indicates the 
nth component of Hi . 

We have examined the symmetry of all possible bilinears 
njMnlli and found out that there are only two sets of bi- 
linears which have the momentum K compatible with the 
\/3 X a/3 ordering. These are given by 

^ / z^n^TiAgni + inlnAgni A 

/ nIriA3ni \ 

^ nIr2A3ni 

V inlraAsHi - l^nlraAgHi / 

In the above X ^3 transforms as the two dimensional £'3 ir- 
reducible representation of the enlarged point group Gp y,. On 
the other hand Xg constitutes a basis of the four dimensional 
Q irreducible representation. Therefore the instability given 
by X has the symmetry consistent with the Vs x Vs orders 
which we have discussed in detail in previous sections. 

Since we are considering an instability from a gapped 
phase, to stabilize the resulting ordered state, the condensa- 
tion energy should be larger than the excitation gap. However, 
because the magnitude of the energy gap reduces as JtJJe in- 
creases, the instability can occur beyond the critical value of 



JtlJe- Finally, since the condition of (Xg,) 7^ does not 
constrain the magnitude of (IIi), (Hi) can have both zero and 
nonzero values. If (Hi ) 7^ with (X^^ ) / 0, we have Z2 spin 
liquid supporting fractionalized quasi-particles and breaking 
the translational symmetry at the same time. This state is 
similar to the Amperean paired stated, which is recently sug- 
gested as a possible ground state of the organic compound 
K-(BEDT-TTF)2Cu2(CN)3. On the other hand, if (Hi) ^ 
while (Xsg ) 7^ 0, we have more conventional phase trans- 
forming as an Ej, irreducible representation. 

In conclusion, the symmetry analysis of the low energy 
fermions near the conduction band minima shows that the 
instability of the SL[0,0,7r] state from the particle-particle 
channel supports valence bond solid phases, which have the 
\/3 X VS unit cell transforming as i?3 irreducible represen- 
tations of the enlarged point group Gp.b- Remember that the 
Je-dimer VBS phase is induced via the monopole condensa- 
tion from the SL[0,0,7r] state for small Jt/ Je limit. Therefore 
we can obtain the valence bond solid ground states both for 
•hi Je ^ 1 and Jt/ Jg ^ 1 limits from the instability of the 
SL[0,0,7r] state. 



C. Projected wave function approach 

We next assess the stability of the SL[0, 0, tt] spin liquid to- 
wards columnar dimer order in the Guzwiller projected state. 
In order to do this, we include a parameter 5t in the mean field 
Hamiltonian which corresponds to strengthening the fermion 
hopping on those bonds which dimerize in the 'classical' 
columnar dimer state, shown in Fig. fTTT a). and Gutzwiller 
project the resulting state. Clearly, if (5t ^ 1, the result- 
ing wave function will be precisely the 'classical' columnar 
dimer pattern. The energy change of the weakly distorted 
state as a function of the distortion parameter 5t serves as a 
measure of the inverse susceptibility of the SL[0, 0, tt] state 
towards columnar dimer order. As seen from Fig. |20l the 
SL[0, 0, tt] state is stable, with a positive inverse susceptibil- 
ity, for Jt/ Je ^ 2, but is unstable, with a negative inverse 
susceptibility for Jt/ Je ^ 2. Further, the optimal 5t ap- 
pears to increase continuously for Jt/ Je ^ 2. This suggests 
that the SL[0, 0, tt] state possibly undergoes a continuous tran- 
sition into a state with \/3 x \/3 columnar dimer order at 
Jt/Je ~ 2.0. We discuss the phase transition more carefully 
in the next section. 



VI. PHASE TRANSITION BETWEEN VBS PHASES 

The results in previous sections show that there is a phase 
transition between the Je -dimer VBS phase and the colum- 
nar 18-site VBS phase with increasing Jt/Je- To describe the 
phase transition between these two VBS phases, we construct 
a Landau-Ginzburg free energy introducing a two-component 
vector (<i>i,$2), which transforms as an E^ irreducible repre- 
sentation of the enlarged point group Gp.b. Since the VBS 
phases are time-reversal invariant, we can use two real num- 
bers, $1 and $2. The Landau-Ginzburg free energy can be 
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FIG. 20: Change in energy of the Gutzwiller projected SL[0, 0, n] 
spin liquid state upon including a distortion St corresponding to in- 
creased fermion hopping amplitude on the dimerized bonds of the 
\/3 X \/3 columnar dimer state shown in Fig. lllf a). For Jt/Je ~ 
1.9, 2.0, the SL[0, 0, n] state is found to be stable against this dis- 
tortion, while it is seen to be unstable for Jt/Je = 2.1, 2.2. (This 
calculation was carried out on a system with 12 x 12 unit cells, i.e., 
with 864 spins. The statistical errors on the computed energy are of 
the order of the symbol size.) 



written using all possible invariants made of (f>i and ^2- In 
particular, it is important to note that there is a third order in- 
variant, which belongs to the Ai irreducible representation of 
the following decomposition. 



E3 (g) E3 (g) E3 = Ai ® Bi ® 3E3 



(26) 



In terms of $1 and $2, the third order invariant is given by 
<I'2(3$i — ^2)- Straightforward extension of the same group 
theoretical analysis to quartic order shows that there is only 
one quartic invariant of + ^2)^. Collecting all invariants 
up to quartic order, the Landau-Ginzburg free energy density 
is written as 

/ = a($2 + + A$2(3$? - + u{<i>l + ^If. (27) 

For convenience we define a complex variable $ as follows, 

$ = $2 - 4*1 = |*|e'^ (28) 

Then the free energy density is given by 

/ = a|$|2-f A|$|3cos(36'). (29) 

Given u > and A > 0, the above mean field free energy 
predicts two different phases separated by a first order tran- 
sition point at a = Q!c = A^/(4/i). For a > ac, we have a 
disordered phase with \^\ = 0. On the other hand, ordered 
phases with |$| ^ and 6 = (n=0,l,2) appear when 
a < ac- The three ordered states describe the three-fold de- 
generate VBS phases with \/3 x \/3 pattern. 
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FIG. 21: Triplet dispersions of valence bond solid ground states 
along high symmetry directions of the Brillouin zone. The dis- 
persions are obtained from the self-consistent bond operator mean 
field theory, (a) Triplet dispersion of the Je-dimer VBS phase for 
Jt/Je = 1. (b) Triplet dispersion of the columnar 18-site VBS phase 
for Jt/Je = 3. 



Considering slow spatial variation of $, the Euclidean 
Landau-Ginzburg effective action is given by 

J d^x\^\d^^f + a\<^\^+u\^\*-\\^fcosi3e)y 

(30) 

The above effective action is nothing but the action for the 
three dimensional Z3 -clock model. Previous Monte Carlo 
simulations suggest that the phase transition associated with 
the Z3 symmetry breaking is weakly first order. ^''^^'^^ The 
wave function numerics in the previous section, however, are 
not inconsistent with this scenario considering the small sys- 
tem size. In addition, the nature of the transition may not have 
been completely settledi^ 



VII. TRIPLON DISPERSION IN THE VBS STATES 

In Fig. I2TI we plot the triplet dispersions of the two VBS 
phases, which are obtained from the bond operator mean field 
theory. The triplet dispersion of the ,/e-dimer VBS state is 
shown in Fig. |2T| (a). Since the unit cell of the Je-dimer 
VBS state is composed of three dimers, we have three triplet 
modes in the spectrum. Here we neglect the fact that each 
triplet particle has three components (x, y, and z) when we 
count the number of bands. Interestingly, the lowest band is 
flat and touches another dispersive band at the Brillouin zone 
center The emergence of this band touching has a topologi- 
cal origin. ^^-^^ The flat band reflects the existence of localized 
eigenstates. In fact, there are two different types of localized 
eigenstates. The first set is given by states which are con- 
fined within dodecagons. Each dodecagon supports a single 
localized eigenstate. In addition, there are "non-contractible" 
loop states constituting the second group of localized eigen- 
states. In contrast to the states confined within dodecagons, 
these loop states extend over the whole lattice system one- 
dimensionally. The topological characteristics of these loop 
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states can be easily understood using the periodic boundary 
condition, under which a two-dimensional system has a torus 
geometry. In this situation, there are two independent loop 
states winding the torus once. Since these states cannot be 
shrunk to points, they are "non-contractible". Counting the 
number of independent localized states carefully, we see that 
the number of degenerate localized eigenstates is larger than 
the number of dimers on the lattice. ■'^•^^ It means that a single 
flat band is not enough to support all independent localized 
eigenstates. Therefore additional degrees of freedom must be 
provided by another band, leading to the band touching. 

The triplet dispersion of the columnar 18-site VBS state is 
displayed in Fig. |2T| (b). We have nine triplet bands which 
are all flat within our mean field approach. The degeneracies 
of the flat bands are given by 3, 1, 2, 2, and 1 counting from 
the bottom to the top bands. In contrast to the case of the 
Jg-dimer VBS phase, the flat structure emerges simply be- 
cause the dimers on expanded links are completely decoupled 
from those on neighboring "perfect" dodecagons. The low- 
est flat band is triply degenerate, which comes from the three 
dimers on expanded links. The remaining six dimers of the 
unit cell lying on "perfect" dodecagons constitute the other 
six bands with higher energies. Therefore we expect that the 
nature of the valence bond solid ground states can be demon- 
strated via neutron scattering experiments measuring triplet 
dispersion spectra. 



Vni. DISCUSSION 



In summary, we have shown that the ground state of the 
nearest neighbor Heisenberg model on the star lattice under- 
goes a phase transition from "the Jg-dimer VBS phase" which 
respects all lattice symmetries to "the columnar 18-site VBS 
phase" which exhibits \/3 x \/3 order with increasing Jt/ Je- 
From the Landau-Ginzburg analysis, this appears to be a con- 
ventional quantum phase transition which is described as the 
thermal transition of the 2 + 1 dimensional Za-clock model. 

If 5 = 1/2 variants of the organic Iron- Acetate magnet can 
be synthesized, they would be particularly good candidates to 
study the phase diagram discussed in this paper since it may 
be possible to pressure tune the ratio Jt/Je significantly in 
such systems. Both VBS states obtained here would exhibit 
a spin gap in uniform susceptibility measurements. We ex- 
pect a direct signature of the 18-site VBS order to appear in 
X-ray scattering or neutron diffraction studies which would 
see a change in the crystal periodicity. Ignoring coupling to 
phonons, the 18-site VBS state should exhibit a thermal tran- 
sition in the universality class of the Za-clock model in D = 2 
dimensions. In addition, the two VBS phases exhibit quite dis- 
tinct behaviors in their triplet excitation spectra as discussed 
above which could be tested using inelastic neutron scattering 
experiments. 
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APPENDIX A: DERIVATION OF THE LOW ENERGY 
EFFECTIVE HAMILTONIANS 

Here we present the details of how we have derived the 
low energy effective Hamiltonians of the spin liquid states dis- 
cussed in Sec. Ill Bl 

1. Effective Hamiltonian for SL [ | , | , tt] 

For the gauge choice depicted in Fig. [Sja), the mean field 
Hamiltonian corresponding to the SL[-|, f, tt] can be written 
in momentum space as, 

HmF = -JtXt E E /L-ff (k)m.n/k.„, (Al) 
k m,n 

in which 
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where 0i=e**^ and Z2=e^^'^^ . m, n are indices for the six 
sites inside a unit cell and \=JeXel JtXt- The indices for the 
spin quantum number are dropped for simplicity. Here we de- 
fine the Fourier transformation via /r^„ = ^ipX^k ^^^^fy^n- 
As described in Fig. |6lb), the conduction (valence) band 
shows the dispersion minimum (maximum) at the momentum 
±Q. The energy eigenvalues of (k) at k=±Q are given by 

Ef ^ y^3/2 + A2 - 1/2V9 + 12A^, 
E^^~E^, (A2) 

where ± indicates the two momentum position ±Q and 1 and 
2 represent the conduction (1) and valence (2) bands, respec- 
tively. To make the analytic treatment of the problem possible 
we focus on the small A limit. Note that the overall spinon 
band structure does not change upon varying A. Expanding 
the energy eigenvalues in powers of A, we get 

Ef = -Ef « ^a2 + 0(A*). 
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The corresponding eigenvectors are 



-i7r/6 1 11 

^ ' ' V3(TTF) 3 ' 3 ' 3 ' 



' ^3(1 + 62) ^ V3 



_^Ae'^/3,-^A,^Ae'2./3}, 
%/3 V3 WS 

{_^Ae-*^/3 ^A, -^Ae-'2V3 
V3 %/3 



— i27r/3 1 11 

„i27r/3 1 1 1 

^3(1 + 62)^ 3 ' 3 ' 3 ' 

^Ae-*''/^ ^A, -^Ae-*2V3} 
V3 W3 

(A3) 

where the superscript T means taking transposition. The 
above eigenvalues and eigenvectors satisfy H{±Q)i'^ 
-^a^a («=1 ™d 2) correctly up to the third order in A. 

Now we want to construct the effective Hamiltonian de- 
scribing the states which have small momentum deviation 
from ±Q, i.e., states with k=±Q+q. We first define Ai?(q) 
= i?(±Q + q)-iJ(±Q). Keeping terms which are first order 
in q and projecting them into the low energy space spanned 
by the eigenvectors (a = 1, 2), we obtain the following 
effective Hamiltonian, 



A 



A2 



i?±(q) = iJ(±Q + q) = j={qxTx + QyTy) + —f=T., 



V3 



Finally, defining the continuum fermion fields using the 
spinon variables as 



pocket centered at the momentum -K. At each momentum 
±K there is a linear band touching as is shown in Fig. lUb). 
The energy eigenvalues of the degenerate states at k=±K are 
given by 



E- = -E^ 



where E'^ {E~) represents the degenerate energy eigenvalue 
at the momentum K (-K). We choose the corresponding 
eigenvectors in the following way. 



p^l'i 1 11 



{-1 - -A2, -1 - -A2. 1 + 

V3(TTP) 3 ' 3 ' 3 



73 ' 73 '73 ^' 



^- {-\-\x\-\-\x\\^\x\ 



73(1 + ^^) 3 



1 Ae-'^2-/^^A,-^Ae'2-/3}, 
\/3 V3 73 

(A5) 

satisfying i?(±K)i/^ =E^v^ {a=\ and 2) correctly up to the 
third order in A. The first order perturbation theory combined 
with the projection into the low energy space spanned by 
{a=\, 2) leads to the following Hamiltonian, 



(^±(q))^ - {E('^?)n/±Q+'.^"'E('^2'=):/±Q+q,4, 

n— 1 n— 1 

(vl/(q)f ^((^+(q)f,(^„(q)n, (A4) 



we arrive at the low energy effective Hamiltonian written in 
Eq. ( fTSl l which is nothing but the Hamiltonian for the massive 
Dirac particles. 



A A^ 

Using the continuum fermion field defined in Eq. ( IA4I ) 
we obtain the low energy effective Hamilonian displayed in 
Eq. (O 



2. Effective Hamiltonian for SL [ f , f , 0] 



The low energy Hamiltonian corresponding to the 
SL[— f , 0] can be obtained following the similar steps used 
to construct the massive Dirac Hamiltonian of the SL[^ , -I , tt] . 
Adopting the flux configuration depicted in Fig.|5jb), we have 
an electron pocket centered at the momentum K and an hole 



3. Effective Hamiltonian for SL[0, 0, 0] 

The mean field band structure of the SL[0, 0, 0] has a flat 
band lying at the fermi energy, which is touching a dispersive 
band at the zone center, i.e., at the momentum T = (0,0). To 
describe the low energy states near the T point, we use the 
degenerate perturbation theory again. However, since the two 
bands touch quadratically, we keep the perturbation expansion 
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up to the quadratic order in momentum. For the SL[0, 0, 0], 
the mean field Hamiltonian is given by 



APPENDIX B: SYMMETRY AND CONTINUUM FIELD OF 

THESLhf , f ,0] STATE 



(A6) 



k m,n 
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We divide H{k) into two pieces such that -ff (k) = Hq + V 
in which iJo = H{\i. = 0). Diagonalization of iJp gives the 
eigenvalues £'o={-l-A,-l-A,-l+A,-l+A,2-A,2+A}. Note that 
there are two pairs of doubly degenerate eigenvalues. Here we 
focus on one of the degenerate eigenvalues u;o=-l+A which is 
lying at the fermi level. We choose the following two degen- 
erate eigenvectors corresponding to wq. 



1 



{2,-1,-1,2,-1,-1}, 



{1^2^ = —{0,1,-1,0,1,-1}, 

Now we introduce the projection operator Po (Pi) which 
projects states into (out of) the low energy space spanned by 
z^i and 1^2- That is, Po=i'i ■ vl + 1^2 ■ i^l '^^'^ A = ^ - Po- We 
also define Vq = PoVPq and Vi = V -Vq. Then iJ(k) = Ho + 
Vo + Vi. The projected Hamiltonian is given by^ 



In this section we show how the continuum fields of the 
SL[— -I, -1,0] state transform under the microscopic symme- 
tries of the lattice. Here we follow the procedures suggested 
by Hermele et al. in Ref. 10. To obtain the necessary in- 
formation we consider a finite system with periodic boundary 
conditions in both ai and 82 directions. In particular, to deter- 
mine how the wave functions at the momentum ±K (we call it 
as the nodal wave functions) transform under the space group 
symmetries of the ansatz, we consider the 3x3 lattice system, 
that is, the system is periodic under the translation by 3ai and 
3a2. Since a unit cell (indexed by a vector R) contains six 
sites labeled by n, the finite system consists of 54 sites. The 
nine points within the Brillouin zone of the finite system con- 
tain the the nodal points ±K and respect all the point group 
symmetries of the ansatz. 

Using the eigenvectors ly^ (a = +,- and a = 1, 2) in Eq. (IA5I) . 
the nodal wave function is given by 



„iaK R 



(Or. 



$a,a(R,n) = - 

Then the continuum field is written as 

4'a,a(q = 0)=^a>;jR,n)/R,„. 

Now we consider a symmetry operation S under which the 
spinons transform as 



Hp„{k)=PoH{k)Po 

^Po[Ho + Vo]Po+PoViPi 



1 



Wo - Hi 



■PiViPo 



(A7) 



which is valid up to the quadratic order in k. The resulting 
Hamiltonian can be written as. 



X 



kxky 



2 A — 3 \ kxky kj. 
We define the continuum fields as 



(V'(k))^ ^ {E(^?):/k."'E('^2^)"/k,n}- (A8) 
n— 1 n— 1 

Finally, combining the above results we obtain the follow- 
ing low energy effective Hamiltonian, 

i?eff=— / -^V^(k)Mk)V'(k), 

meff J (2ttY 



in which 



/leif(k) ={ki + A:;)ro - {ki - k;)T, - 2k^kyT^ 



In particular we consider the following symmetry opera- 
tions, 

(i) The ai translation (Ti); 

Tl :/R,n,cr — > Gti (R, 'ri)/R+ai,n,(T, 

(ii) The 82 translation (T2); 

T2 ■fR,n,cr (R, ?^)/R+a2,n,(Tj 

(iii) The ^ rotation (02^^/3); 

C27r/3 ■fR,n,a — > Gca^/g (R, ?^)/c2„/3(R,n),<T, 

(iv) The y reflection (Ry); 

-^y •f{x,y),n.{T ^ ^-Ry (R5 — y),7n,o'5 

Where R=(x,y) and (n,m)e{(l, 4), (2, 6), (3, 5), (4, 1), (5, 3), (6, 2)}. 

(v) Time-reversal and inversion (T • /); 

T ■ I :/R,ri,o- (*f2)<j,o"/-R,(n+3),cr') 

in which (nH-3) represents the remainder when it is divided by 
six. 
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(vi) Charge conjugation (C*); 

in which = 1 for i = 1, 2, 3 and -1 otherwise. 

For each of the symmetry operation S, the matrix represen- 
tation of the symmetry S is defined as 



(5)5(i)a ^ G5(i). 

The action of the symmetry operation S on the nodal wave 
function is given by S^a=Cab^b- Now a and b denote the 
nodal and the two-component Dirac indices collectively. Fi- 
nally, the transformation of the nodal wave function reflects 
the action of S on the continuum field such as 



5 : *a -> cl^^b- 

The transformation rule of the continuum field under all the 
above symmetry operations is summarized in Eq. ([TtI i. 



APPENDIX C: EXPRESSIONS OF THE MATRICES As(Tli) 
AND Bs(n») 

Here we present the expressions of the matrices ^5(11;) 
and Bs{^i) {S=Ta^, Tq^, Ry, and Cj^), which are defined in 
Sec. lVBI At first As{^i) are given by 
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Similarly for ^5(112), 
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In the case of Bs{^i 
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Similarly for Bs(f72), 
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